Repository navigation
Add io_whole_file benchmark and an opt-in io group - #498
jayzuccarelli wants to merge 3 commits into
Conversation
Add an "io" benchmark group, resolved through the "io" tag and excluded from the default group, so I/O benchmarks only run with "-b io". io_whole_file reads and writes a whole file in a single call: open().read()/write() in binary mode plus Path.read_bytes/write_bytes at 4 KiB and 1 MiB, and Path.read_text/write_text at 1 MiB (6 results). Part of python#399.
|
Can you explore ways to make this measure
For the largefiles that is probably a second opt in (cpython's core tests do that), although needs some design work |
|
Sure. Rough plan:
Would you rather keep this PR as the cached baseline and do uncached and largefiles as follow-up PRs, one per benchmark like before? |
|
I think building out the core primitives and how understandable the results are are important in the first PR. Need to do it with straightforward cases to make sure it's working right before we do more complicated ones. |
Add io_whole_file_read_bytes_{4k,1m}_uncached. Before each timed
open().read(), the file's pages are dropped from the page cache with
os.posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED), outside the timed region,
so the read has to wait for the storage device. The read file is fsync'ed
once in setup because DONTNEED only drops clean pages.
The uncached results are skipped where os.posix_fadvise is missing (macOS,
Windows). Writes stay cached-only: an uncached write would need an fsync in
the timed region, which measures the device's flush path, not Python.
|
Added uncached reads as |
First of three PRs for #399, one benchmark each as suggested there. This one adds the shared piece (an
iogroup that is not part ofdefault) together with the first benchmark that uses it.What it measures
io_whole_filereads and writes a whole file in a single call, which is the pattern gh-120754 optimized:Each result measures a single API:
read_bytes_{4k,1m}:open(path, "rb").read()write_bytes_{4k,1m}:open(path, "wb").write(data)read_text_1m/write_text_1m:Path.read_text()/Path.write_text(), utf-8, ASCII contentread_bytes_{4k,1m}_uncached:open(path, "rb").read()with the file's pages dropped from the page cache firstPath.read_bytes()/write_bytes()are thin wrappers over the sameopen()calls, so they are not measured separately.8 results on Linux (6 where
os.posix_fadviseis missing). 4 KiB is where per-call overhead and the number of system calls dominate; 1 MiB is where buffer allocation and copying dominate. I had an 8 MiB size too, but it scaled linearly from 1 MiB and sits closer to dirty page throttling, so I dropped it to keep the result count small.Apart from the uncached reads below, everything runs in a fresh temp dir with a warm page cache and no fsync, so it measures Python's I/O stack and syscall overhead rather than the disk. Writes always create a new file (unlinked outside the timed region): truncating and rewriting an existing file on ext4 triggers writeback on close (
auto_da_alloc), which made a 16 MiB rewrite take ~490 ms +- 220 ms instead of ~30 ms. The run recordsio.DEFAULT_BUFFER_SIZEin the metadata.Uncached reads:
read_bytes_{4k,1m}_uncachedtime the sameopen(path, "rb").read()with the file's pages dropped from the page cache before each read (os.posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED), outside the timed region; the file is fsync'ed once in setup since DONTNEED only drops clean pages, no root needed).fincoreshows 0 resident pages after every drop. On a SATA SSD (ext4, 3.14) 4 KiB goes from ~17 us cached to ~204 us uncached and 1 MiB from ~148 us to ~3.4 ms. They mostly track the storage device, so they are noisier and named apart from the cached results. They are skipped whereos.posix_fadviseis missing (macOS, Windows; macOSF_NOCACHEdoesn't evict already-cached pages). Writes stay cached only, since an uncached write needs an fsync in the timed region, which measures the device's flush path rather than Python. Large files are left for a follow-up.Setup allocates and frees one 8 MiB block before anything is measured. On glibc that raises malloc's dynamic mmap/trim thresholds above the 1 MiB working set; without it
read_text_1m(which frees two 1 MiB blocks per call) sat right at the trim threshold and, depending on heap history, flipped between ~430 us and ~1.3 ms on the same interpreter because the heap top was handed back to the kernel and faulted in again every call.Why opt-in
Per the issue, I/O benchmarks shouldn't run by default. The benchmark is tagged
io,[group io]picks it up through the tag, and[group default]excludes it by name (a-iogroup exclusion is silently ignored by the manifest parser, so it has to name the benchmark).pyperformance listis unchanged at 100 benchmarks;-b ioselects it.Sample timings (
pyperformance run -b io_whole_file --fast, one untuned Linux box, ext4 /tmp)How to run
Tests
python -m pyperformance.testspasses (it runs-b all, so the io benchmarks execute there too)pyperformance listunchanged at 100,list -b iois justio_whole_fileos.posix_fadviseremoved (6 cached results, no crash)__version__too if you want that in the same PR, as Add bm_base64 covering common base64 module APIs #447 did.Part of #399.
AI disclosure: I used Claude Code to write the benchmark and docs; I reviewed the code and ran it.